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I  INTRODUCTION 


In  the  language  of  Dixon  and  Mood  (1948)  a  sensitivity  testing  environment  may  be 
described  as  follows.  The  sensitivity  of  explosives  to  shock  is  tested  by  dropping  a 
weight  on  the  explosive  from  various  heights  above  it.  The  result  is  either  a  ’’response" 
(explosion)  or  a  '’nonresponse”.  It  is  assumed  that  there  exists  for  each  explosive  some 
critical  height,  above  which  the  explosive  will  always  detonate.  Critical  height  is  then  a 
continuous  variable  which  cannot  be  measured  directly  but  rather  only  through 
observing  response  or  nonresponse  for  the  various  levels  of  height.  Commonly  assumed 
is  that  a  continuous  monotone  increasing  function  F(x)  represents  the  probability  of 
observing  a  response  for  each  height  x.  The  ultimate  goal  of  sensitivity  testing  is  to 
adequately  describe  some  interval  of  this  function.  In  this  manner,  the  sensitivity  of  the 
explosive  to  levels  of  shock  is  modeled. 

Since  the  Up  and  Down  Method  (1948)  appeared  in  the  literature,  several  other 
data  collection  and  estimation  procedures  have  been  proposed  including  Robbins-Monro 
(1951),  Langlie  (1962),  Wet heri  11  (1963),  and  Wu  (1985).  Many  of  these  procedures  have 
been  subjected  to  extensive  study  and  comparison.  However,  much  of  the  work  is 
concerned  with  asymptotic  properties,  for  example.  Chung  (1954)  and  Hodges  and 
Lehmann  (1955)  concerning  the  Stochastic  Approximation  Method  of  Robbins  and 
Monro.  Unfortunately.  Langlie's  procedure,  the  Army  standard,  has  been  ignored  to  a 
great  extent  in  the  literature.  It  is  the  purpose  of  this  study  to  compare  the 
performance  of  these  and  more  recent  procedures  under  the  constraints  imposed  in  a 
field  test  environment. 

In  consideration  of  the  Army  test  environment,  a  more  practical  accounting  of  test 
condition  effects  must  be  made  than  is  given  in  many  of  the  previous  studies.  Because  of 
the  cost  of  the  munitions  used  for  testing,  only  small  samples  are  possible,  and  hence 
asymptotic  results  add  little  to  performance  understanding.  In  some  studies  maximum 
likelihood  estimates  are  compared  to  others,  but  only  when  the  maximum  likelihood 
estimate  exists  uniquely.  Evaluation  of  techniques  in  our  expensive  test  environment 
must  consider  back-up  estimation  procedures  when  the  necessary  conditions  for  the 
intended  estimator  are  not  satisfied.  Also  for  penetrator-against-plate  testing,  a 
specified  stress  (velocity  of  the  penetrator)  cannot  be  guaranteed.  For  instance,  one 
may  intend  to  fire  a  round  at  2500  fps  and  then  assess  his  results  (penetration  or 
nonpenetration)  so  that  a  next  stress  level  (Next  Stress)  can  be  chosen.  However,  since 
the  velocity  of  a  round  cannot  be  guaranteed  by  varying  the  propellant  charge  weight, 
there  is  white  noise  associated  with  the  actual  value  of  the  stress  level  considered.  It  is 
important  to  examine  how  sensitive  the  results  are  to  this  uncontrollable  white 
noise.  The  feasibility  of  maximum  likelihood  estimation  under  these  conditions  is 
discussed  by  Golub  and  Grubbs  (1956). 

The  overriding  consideration  in  preparation  of  this  study  was  to  create  a 
representative  slice  of  real-life  situations  wherein  data  collection  and  estimation 
procedures  could  be  compared.  Several  interesting  small  sample  studies  were  drawn 
upon  for  this  purpose.  Dixon  (1965)  and  Hampton  (1967)  look  at  the  effect  of 
misplacement  error  (poor  choice  in  starting  value)  with  regard  to  the  sequential  Up  and 
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Down  Method.  Dixon  also  looks  at  the  effect  of  a  poor  guesstimate  of  the  response 
distribution  standard  deviation:  this  controls  the  size  of  change  in  stress  made  for  the 
next  design  point.  Davis  (1969)  considers  sensitivity  to  response  distribution 
assumptions  in  addition  to  the  above  two  errors  for  several  procedures  proposed  at  that 
time.  Sample  sizes  considered  in  all  three  were  relatively  small.  We  will  use  these 
devices  along  with  some  additional  ones.  Additional  devices  include  the  use  of  white 
noise  in  the  stress  levels  and  the  use  of  backup  estimation  procedures  when  the  intended 
estimation  fails.  Also,  some  specific  sample  sizes  are  considered.  These  are  sample  sizes 
recommended  recently  by  standard  operating  procedures.  This,  intended  as  a  practical 
study,  considers  actual  data  as  well  as  Monte  Carlo  simulations.  All  designs  and 
estimation  procedures  will  be  applied  to  an  empirical  distribution  formed  from  testing 
performed  at  Aberdeen  Proving  Ground  (APG).  Finally,  the  standard  test  strategy. 
Langlie's  One  Shot  Test  Strategy,  will  be  compared  to  more  recent  designs  proposed  in 
the  literature  such  as  the  Delayed  Robbins-Monro  of  Cochran  and  Davis  (1964)  and  the 
Estimated  Quantal  Response  Curve  of  Wu  (1985).  With  these  considerations  in  mind 
procedures  are  compared  according  to  their  ability  to  estimate  the  ballistic  limit  V50.  the 
velocity  at  which  half  of  the  projectiles  would  be  expected  to  penetrate  the  armor,  i.e.. 
the  median  of  the  response  distribution  F(x). 

Because  of  the  number  of  factors  which  comprise  the  real-life  scenario,  it  is 
appropriate  to  introduce  all  of  the  factor  levels  in  a  diagram.  See  Figure  1.  This  is 
intended  as  an  aid  to  the  readers  so  that  they  can  more  easily  keep  the  sections  that 
follow  in  proper  perspective.  In  those  sections,  elements  of  the  diagram  are  discussed  in 
more  detail. 

C.  DATA  COLLECTION  AND  ESTL\L\TION  PROCEDURES 

Data  collection  and  estimation  should  be  considered  separate  processes.  It  is  true 
in  the  case  of  some  test  strategies  that  the  next  design  point  chosen  is  also  the  estimate 
of  the  target  quantile.  This  of  course  need  not  be  the  case.  With  each  data  collection 
technique,  three  estimation  procedures  are  employed:  NMLE,  A\R,  Next 
Stress.  NMLE  is  the  maximum  likelihood  estimator  of  V50  when  a  normal  distribution 
is  assumed  for  the  distribution  of  critical  velocity.  AYR  is  an  averaging  of  stress  levels 
thought  to  be  near  the  median.  Next  Stress  will  vary  depending  on  which  data 
collection  procedure  is  being  employed.  For  instance  in  the  case  of  the  Langlie 
procedure  it  is  the  next  design  point  of  an  averaging  algorithm,  whereas  for  Wu’s 
technique  it  is  intended  as  the  logit-based  maximum  likelihood  estimate  computed  using 
the  information  gathered  up  until  that  point. 

Five  designs  will  be  examined  in  this  study:  Langlie,  Delayed  Robbins-Monro 
(DRM),  Adaptive  Robbins-Monro  (ARM),  Estimated  Quantal  Response  Curve-Delayed 
Robbins-Monro  (EQRC-DRM),  and  Estimated  Quantal  Response  Curve-Adaptive 
Robbins-Monro  (EQRC-ARM)  The  first  is  the  method  currently  used  by  the  U  S. 
Department  of  the  Army  in  penetrator-against-plate  testing.  The  second  is  a  slight 
adjustment  of  a  well  known  and  widely  used  technique.  The  final  three  are  fairly  recent 
attempts 
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at  improving  on  the  Robbins- Monro  (RM)  procedure.  Details  for  each  design  interfaced 
with  the  three  estimation  procedures  follow 


1.  Lundies  One  Shot  Test  Strategy 

Langlie's  One  Shot  Test  Strategy  was  introduced  in  1962.  The  experimenter  is 
asked  to  make  a  guesstimate  of  the  location  of  the  median  of  the  response 
distribution.  In  practice,  the  actual  questions  are  "At  what  stress  can  we  be  reasonable 
certain  that  a  response  will  always  occur1"  and  "At  what  stress  can  we  be  reasonably 
certain  that  a  response  will  never  occur?".  The  two  stresses  are  referred  t<i  as  the  upper 
and  lower  gates,  respectively.  All  design  points  using  the  Langlie  strategy  must  reside 
within  these  two  gates.  The  first  design  point,  the  experimenter's  estimate  of  the 
median,  is  taken  to  be  the  average  of  the  upper  and  lower  gates.  Subsequent  points 
xn^!  are  taken  to  be  the  average  of  xn  and  xr  ;.  where  i  is  the  first  time  in  searching 
back  in  the  data  that  the  number  of  responses  equals  the  number  of  nonresponses 
between,  inclusively.  xn  and  xn_j.  If  no  such  i  exists,  xn  is  averaged  with  the  upper  gate 
for  xn.  nonresponse,  and  the  lower  gate  for  xn.  response.  The  tendency  of  experimenters 
in  the  past  has  been  to  pick  extremely  wide  gates,  equivalent  to  \  50  ±  5er  or  more, 
because  of  the  gate  restriction  on  possible  design  points. 


The  intended  estimation  for  the  Langlie  method  is  maximum  likelihood  with  an 
assumed  normal  response  distribution  for  the  critical  velocity.  For  a  response  function 
P  with  mean  \i  standard  deviation  <7  and  stress  levels  x,,  the  likelihood  function 
-discussed  by  Dixon  and  Mood  (1948)  and  Golub  and  Grubbs  (1956)  is  given  by 


P  =  p;d‘  q;1  d‘  with  d;  =  (0.  1)  =  (nonresponse,  response) 
i 


t, 

and  p|=  J 

-00 


dt  —  1-q', 


where  tj  = 


(1) 


The  likelihood  equations  formed  are  solved  iteratively  using  the  Newton-Raphson 
procedure.  There  are  several  algorithms  available  which  perform  this  procedure.  A 
problem  with  many  is  slow  convergence  or  failure  to  converge  in  many  practical 
situations.  The  algorithm  of  DiDonato  and  Jarnagin  (1972)  as  prepared  for  local 
computing  facilities  by  McKaig  and  Thomas  (1983)  was  used  in  this  study.  The  claimed 
advantage  is  guaranteed  global  convergence  to  ”best"  estimates  when  they  exist 
uniquely,  even  for  poor  initial  starting  values.  This  claim  was  based  on  extensive  testing 
performed  by  the  original  authors.  Unique  estimates  exist  for  data  structured  such  that 

(X  mim  X  maj(  )  PI  min>  x  max)  7^  ^  (^) 

where  x+  and  x~  indicate  stress  levels  at  which  a  response  and  a  nonresponse  was 
observed,  respectively.  This  situation  is  commonly  referred  to  as  a  zone  of  ’’mixed" 
results.  An  additional  requirement  for  the  algorithm  is  that  there  be  a  minimum  of  two 
nonresponses  and  responses. 
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A\"R.  an  alternative  estimation  procedure,  is  the  average  of  stress  level-  pr  vidinc 
the  k  lowest  responses  and  the  k  highest  nonresponses.  The  value  k  is  generally  taken 
to  be  3.  when  possibie.  but  no  less  than  2  is  considered  here.  AVR  is  used  currently  by 
the  Combat  Systems  Test  Activity  (C'STA)  of  APG  when  their  algorithm  for  maximum 
likelihood  estimation  fails  to  converge  or  when  there  is  no  zone  of  "mixed"  results.  Thi- 
method  could  be  considered  an  extension  to  comments  made  by  Brownlee.  Hodg**s.  and 
Rosenblatt  in  a  1953  paper.  There  they  suggest  averaging  stress  levels  which  reasonably 
hold  pertinent  information  for  the  estimate  of  the  median  using  data  resulting  from  I’p 
and  Down  testing.  Wetherill  (1963)  cautions  that  before  an  average  of  stress  levels  is 
made,  consideration  should  be  given  to  the  type  of  response  elicited  by  these  stress 
levels.  The  A\"R  method  mentioned  here  does  consider  type  of  response.  It  is 
supported  somewhat  in  the  knowledge  that  a  convergent  type  of  algorithm,  such  as  the 
Langlie,  should  provide  one  with  stresses  about  the  median. 

The  final  estimation  procedure  considered  is  the  Next  Stress.  In  the  case  of  the 
Langlie  strategy,  it  is  not  expected  to  be  a  very  good  estimator,  since  it  is  very  possible 
that  this  final  estimate  may  consist  of  the  average  between  a  design  point  and  a  gate. 

In  order  to  compare  designs  in  a  real-life  environment,  reasonable  estimation  must 
be  possible  for  each  set  of  test  conditions  considered.  If  the  WILE  is  far  and  away  the 
best  estimate  of  V50  when  it  exists  uniquely  but  only  exists  uniquely  oOGf  of  the  time, 
much  testing  could  go  wasted.  In  practice,  an  estimate  is  usually  attempted  rather  than 
waste  the  money  and  efforts  expended  during  the  test,  either  for  direct  use  or  for 
designing  another  experiment.  For  this  reason,  the  estimation  procedures  actually  used 
will  be  combinations  of  the  above  three  methods,  thus  insuring  that  some  estimate  will 
always  be  made  regardless  of  test  conditions.  We  refer  to  these  as  practical  estimation 
schemes.  The  practical  estimation  schemes  used  in  connection  with  the  Langlie  will  be 
as  follows. 

Next  Stress  is  the  next  design  point  of  the  algorithm.  There  is  no  need  for  a  back¬ 
up  procedure  if  Next  Stress  is  being  considered,  since  a  next  design  point  will  always  be 
possible  using  the  algorithm. 

For  AVR.  it  is  thought  that  a  minimum  of  two  responses  and  nonresponses  should 
be  considered  when  averaging  the  stress  levels.  Thus,  if  there  is  only  one  response  or 
nonresponse,  a  back-up  procedure  will  be  used.  We  have  chosen  Next  Stress  as  this 
back  up  procedure.  Hereafter,  when  considering  the  technique  AVR,  we  are  referring  to 
th  is  two- lev  el  process. 

In  terms  of  maximum  likelihood  estimation,  the  following  three-level  estimation 
process  will  be  used.  First,  use  the  WILE's  if  they  exist  uniquely  within  the  range  of 
the  data.  Second,  when  the  WILE's  do  not  exist  uniquely,  use  the  AVR  technique. 
Occasionally  the  WILE's  will  exist  uniquely  but  outside  the  data  range.  In  our 
experience  this  commonly  results  from  a  high  concentration  of  observations  taken  near  a 
gate  where,  for  example,  most  are  nonresponse  and  a  small  percentage  are 
response.  The  WILE’s  are  then  some  value  above  the  gate.  Many  times  this  estimate 
is  greatly  inflated  because  testing  has  not  been  performed  in  the  correct  region,  that  is 


not  near  the  median.  It  is  then  necessary  to  consider  a  back-up  estimation 
procedure.  Since  the  reason  for  this  situation  is  a  poor  choice  of  design  levels.  AAR  is 
not  a  reasonable  choice.  In  fact,  a  reasonable  choice  may  not  exist  given  the  restriction 
of  the  Langlie  gates.  We  chose  to  make  this  estimate  Next  Stress,  which  in  this 
situation  will  approximate  the  gate.  We  realize  that  this  convention  is  not  ideal,  but 
first  remember  that  in  our  experience  it  has  only  occurred  in  the  above  undesirable 
situation.  Secondly,  the  primary  function  of  these  designs  is  to  provide  observations 
giving  rise  tc  good  estimation.  Failure  to  do  so  under  varying  conditions  is  simply  a 
penalty  of  using  that  design.  The  number  of  necessary  Next  Stress  gate  approximations 
will  be  recorded  and  considered  when  we  do  the  final  design  comparison.  Hereafter, 
when  considering  the  NMLE  estimator,  we  are  referring  to  this  three  level  process. 

Th  ese  three  practical  estimation  procedures  will  be  used  for  each  of  the  five 
designs.  The  only  difference  in  practical  estimation  among  the  designs  will  occur  with 
the  designs'  choice  of  Next  Stress.  This  amounts  to  the  comparison  of  15  possible  data 
collection  and  estimation  procedures. 

2.  Delayed  Robbins-Monro  Stochastic  Approximation  Method  (DRM) 

The  Stochastic  Approximation  Method  was  proposed  by  Robbins  and  Monro  in 
1051.  The  DRM  is  essentially  this  same  procedure  with  an  adjustment  to  help  the 
design  sample  from  the  correct  region.  So  that  the  experimenter  will  be  required  to 
think  of  the  sensitivity  test  as  he  is  accustomed,  the  starting  position  for  the  DRM  and 
the  magnitude  (to  be  discussed  later)  of  the  constant  c  will  be  be  drawn  from  the  same 
information  that  the  experimenter  commonly  provides  in  order  to  use  the  Langlie 
procedure.  Select  a  starting  value  in  the  same  manner  as  for  the  Langlie  strategy. 
Cochran  and  Davis  (1961).  and  Davis  (I960)  suggest  making  design  points,  prior  to  a 
reversal. 

xn+l  =  xn-«(.vn-  -5)-  (4) 

with  yn,  a  (0,1)  outcome  and  with  c  a  constant.  Reversal  is  the  occurrence  of  (response, 
nonresponse)  or  (nonresponse,  response)  in  succession.  Subsequent  design  points  will  be 
chosen  according  to  the  usual  RM  method  by 

Xn+1  =  Xn  "  n-k+T{yn  _  '5)’  (0) 

where  k  is  the  first  sample  number  corresponding  to  the  first  reversal.  The  results  of 
Davis’  small  sample  study  show  the  DRM  to  be  one  of  the  best  performers  of  the  designs 
considered. 

3.  Adaptive  Robbins-Monro  (ARM) 

The  ARM  is  discussed  by  Anbar  (1978).  Wu  (1985)  suggests  an  alternative 
truncation  rule  which  will  be  used  here.  As  with  Langlie  and  DRM,  the  .ARM  will 
request  the  experimenter's  guesstimate  of  reasonable  upper  and  lower  gates.  Design 
points  are  picked  as 

xn+l  =x„-c(.vn--5)  <6) 
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until  a  reversal  occurs.  It  has  been  shown  through  the  work  of  Chung  (195-4)  and 
Hodges  and  Lehmann  (1955)  that  the  optimum  c  is  (F*  ( . 5 ) ) “ 1 ,  where  F  is  the  response 
function  for  the  critical  velocity.  Use  of  the  theoretical  result  is  made  when  choosing 
subsequent  design  points  by 

K1 

Xn+1  =  Xn  -  ()'n  -  -5).  (7) 


where  3  is  the  regression  slope  estimate  given  by 


i=l 


-  xn)= 

i=l 


(8) 


and  k.  as  before,  is  the  sample  number  marking  the  first  reversal.  Because  of  the 
variability  commonly  associated  with  3.  the  ARM  actually  only  uses  3  provided  that  it 
falls  within  a  specific  range.  Wu  suggests  truncating  rather  than  3.  This  yields  a 
next  design  point  of 


Xn  +  1  Xn  n -k-f- 1  (yn 


•  ol 


(9) 


where  dn  is  given  by 


dn  =  max(min(  jn'\  c).  6) 


(10) 


with  c>(5> 0  and  c  to  be  discussed  later. 


4.  Estimated  Ouantal  Response  Curve  (EQRC) 

The  EQRC  proposed  by  Wu  (1985)  is  a  general  technique  by  which  the  parameters 
of  an  assumed  response  function  can  be  estimated.  In  this  study  the  logit  response 
function  is  assumed  and  maximum  likelihood  estimation  is  employed.  The  next  design 
point  is  taken  to  be  the  logit-MLE  (LMLE)  when  it  exists  uniquely,  based  on  the  data 
up  until  that  point.  This  requires  that  data  be  collected  in  an  alternate  fashion  until 
the  LMLE  can  be  used.  Based  on  Wu’s  suggestion,  we  choose  to  use  two  different  base 
designs'.  DRN1,  ARM.  EQRC-DRM  and  EQRC'-ARM  are  treated  as  separate  data 
collection  procedures. 

Unique  existence  of  the  LMLE,  Silvapulle  (1981),  is  guaranteed  by  a  zone  of 
’’mixed"’  results.  Even  when  it  exists  uniquely,  the  estimates  can  vary  greatly  for  small 
samp.<_  sizes.  For  this  reason,  the  base  design  is  used  up  to  and  including  sample  point 
six  or  whenever  the  LMLE  exists  uniquely,  which  ever  comes  later. 

The  logit  distribution  is  given  by 

F(x  |  0)  —  - I—  X  >  0.  0  =  (a,  X)  .  (11) 


where  a  is  the  desired  estimate  for  V-0.  The  maximum  likelihood  estimates  can  be 
obtained  by  solving 


n 

NT 


F(\'j  |  a,  X)  =  E>\ 


Exi  F(Xi  |  ct,  X)  =  E>'ixi 

i=l  i=l 


\  L~l 


(13) 


in  an  iterative  fashion.  In  practice,  it  is  doubtful  that  field  experimenters  will  draw  on 
an  iterative  solution  to  these  equations  for  their  next  stress  level.  Realizing  this,  Wu 
(1985)  suggests  using  an  approximate  solution  to  these  equations.  The  approximation 


1 

l+e_t 


(11) 


is  substituted  in  the  above  equations  for  the  logistic  distribution.  F(x|0),  defined  in 
Equation  11.  Then  the  next  design  point  q  is  a  weighted  average  of  the  stress  levels 
expressed  by  as 


a, 


n  n  i  n  n  i 

i=l  i=l  ^  i==1  1=1 

n  n  i  n  n  i 


(15) 


The  accuracy  of  this  approximation  varies  with  t=X(x-or).  Cox  (1970)  determines 

that  the  approximation  will  be  within  .07  of - - -  for  values  of  t  in  the  interval 

I  +  e-t 

(-3,  3).  The  corresponding  rule,  stated  in  terms  of  stimulus  location,  insures  .07  accuracy 
for  F(x|0)  for  stimulus  levels,  x,  within  1.65  standard  deviations  of  the  mean.  For 
stimulus  levels  gathered  outside  that  interval,  for  example,  at  two,  three,  or  four 
standard  deviations  from  the  mean,  the  accuracy  degrades  quickly  to  within  .13.  .41, 
and  .71,  respectively.  This  fact  implies  that  the  approximation  is  poor  when  the 
function  is  evaluated  at  stimulus  levels  distant  from  the  mean.  However,  this  problem 
is  not  anticipated  to  be  serious  for  two  reasons.  First,  the  convergent  nature  of  the 
sequential  procedure  will  concentrate  most  stimulus  levels  about  the  mean.  Second,  the 
weighted  average,  an,  weights  more  heavily  those  stimulus  levels  close  to  the 
mean.  Therefore,  the  approximation  should  perform  reasonably  well  in  this  setting,  and 
no  attempt  will  be  made  to  identify  or  eliminate,  from  estimation,  extreme  levels  of  the 
stimulus. 


Besides  insuring  that  the  LMLE  be  used  no  sooner  than  seven  samples  into  the 
design,  it  is  also  desirable  to  truncate  the  design  thus  limiting  moves  which  are  too 
large.  With  this  additional  consideration  the  data  collection  procedure  is  as  follows  for 
both  EQRC-DRM  and  EQRC-ARM. 


Sample  using  the  DRM  or  ARM  procedure  until  the  LMLE  exists  uniquely  and 
sample  six  has  been  taken.  Subsequent  design  points  are  taken  to  be 


d  * 


Xn+1  = 


n-k+1 


(Yn  -  .5), 


(16) 
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where  dn*  is  determined  by 


(17) 


dn*  =  max(min(dn  c),  6), 
with  c  >  <5  >  0.  The  value  dn  is  the  solution  of 

dn 

^n+i  —  r—r(>n  ~  •**)>  (1^) 

n-k+1 

where  xn+1  is  the  LMLE  d  of  V50.  and  k  is  the  sample  point  marking  the  first  reversal. 

m.  QUESTIONS  OF  INTEREST 

What  effect  will  sample  size  have  on  the  distribution  of  V50?  In  practice,  for 
penetrator-against-plate  testing,  sample  sizes  are  commonly  less  than  15,  a  fairly 
restrictive  sample  for  sensitivity  testing.  We  would  like  to  see  what  happens  to  V50  as 
we  vary  small  sample  sizes. 

What  is  the  effect  of  noise  on  the  distribution  of  V50?  With  the  exception  of  Golub 
and  Grubbs  (1956),  all  of  the  work  to  date  on  these  sensitivity  designs  assume  that  the 
next  design  point  intended  by  the  design  can  be  attained.  In  penetrator-against-plate 
testing  this  is  not  the  case.  We  wish  to  examine  these  procedures  for  their  sensitivity  to 
different  noise  environments. 

What  is  the  effect  of  misplacement  error  on  the  distribution  of  V50?  The 
experimenter's  guesstimate  of  the  quantile  location  for  a  newly  developed  item  can  often 
be  very  much  in  error.  In  some  situations,  the  median  of  the  response  distribution  has 
been  guessed  with  more  than  3 a  error,  as  shown  by  further  experimentation.  The 
parameter  a  is  taken  to  be  the  standard  deviation  of  the  response  distribution.  What 
procedure  will  smooth  over  this  poor  initial  guesstimate? 

What  is  the  effect  of  error  in  the  guesstimate  of  the  response  distribution  standard 
deviation?  Commonly,  gates  are  established  at  V60  ±  3a.  The  concern  is  to  insure  that 
the  median  is  within  the  gates  since  Langlies  procedure  makes  no  allowance  for  gate 
adjustment  while  testing.  It  sometimes  results  in  ridiculous  gate  setting,  such  as 
V50  ±  10cr,  which  results  in  several  of  the  first  few  rounds  being  wasted.  Each  design 
and  combined  estimation  procedure  will  be  assessed  for  its  ability  to  rebound  from 
mildly  poor  guesstimates. 

How  do  the  procedures  respond  to  different  underlying  response  distributions?  If  a 
parametric  assumption  is  made,  it  has  usually  been  probit  or  logit  in  the  literature. 
Davis  (1969)  considers  normal,  uniform,  and  exponential  response  functions  in  his  small 
sample  study.  His  results  indicate  robustness  to  response  distribution  for  each  of  the 
estimation  procedures  considered.  We  will  look  at  this  question  again  for  completeness 
in  light  of  our  additional  considerations. 
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Which  estimation  procedure  fares  best  over  the  wide  range  of  test  conditions  for 
each  of  the  designs  individually’  There  is  no  reason  to  limit  ourselves  to  the  intended 
estimation  associated  with  each  design,  especially  considering  that  they  make  different 
distribution  assumptions.  We  would  like  to  pick  an  estimator  which  performs  fairly  well 
over  a  range  of  real-life  representative  conditions. 

Which  design  and  estimation  procedure  fares  best  over  all?  This  is  the  most 
important  question  to  be  addressed  in  this  analysis.  We  would  like  to  conclude  this 
study  with  a  sound  recommendation  for  data  collection  and  estimation,  yielding 
reasonable  estimates  over  a  wide  range  of  real-life  conditions.  We  will  do  this  bv 
comparing  the  best  estimation  results  of  each  design.  Also  compared  are  the  intended 
estimation  results  for  each  design. 


IV.  DESIGN  CONSIDERATIONS 


1.  Factors 


Five  response  functions  were  considered  for  the  distribution  of  critical  velocity: 
normal,  uniform,  exponential,  Cauchy,  empirical.  The  first  three  distributions  were 
given  a  median  0  and  a  standard  deviation  1.  The  normal  was  chosen  because  the 
Langlie  uses  this  assumption  for  estimation,  and,  similarly,  a  logit  assumption  is  made 
for  EQRC.  The  uniform  was  chosen  as  an  extreme  case  of  the  symmetric 
distributions.  It's  flat  density  should  slow  convergence  of  the  sequential  designs.  The 
exponential  was  chosen  since  it  is  a  common  asymmetric  distribution  and  we  wanted  to 
see  what  problems  asymmetry  causes  in  data  collection  and  estimation.  The  Cauchy 
distribution  was  chosen  for  its  heavy  tails.  Parameters  were  set  so  that  the  quartiles 
were  the  same  as  for  a  normal(0,l)  distribution.  We  wanted  to  see  if  those  heavy  tails 
adversely  affected  the  estimation  of  the  median.  The  main  observations  and  conclusions 
are  to  be  drawn  from  these  four  distributions. 

The  empirical  distribution  referred  to  was  formed  using  real  penetrator-against- 
plate  data.  It  had  an  estimated  median  of  1-37.6  and  an  estimated  standard  deviation  of 
40.8.  The  data  consisted  of  79  stress  levels  with  an  associated  response  or  nonresponse. 
Using  the  "method  of  reversals"  as  outlined  in  Rothman,  Alexander,  and  Zimmerman 
(1965)  we  formed  the  empirical  distribution  function.  For  easier  simulation,  we  then 
smoothed  the  step  function  with  a  third  degree  polynomial,  and  forced  the  right  limit  to 
1  and  the  left  limit  to  0  in  order  to  make  it  an  actual  distribution  function. 

The  sample  sizes  (SS)  considered  are  9,  12,  and  15.  Until  recently,  CSTA's  sample 
size  strategy  was  to  fire  as  many  as  15  rounds  in  hopes  of  getting  at  least  12 
observations;  erratic  flight  necessitates  invalidation  of  some  rounds.  In  1983,  because  of 
excessive  costs,  samples  were  cut  to  as  low  as  9  in  the  acceptance  testing  of  some 
penetrators.  It  is  important  to  attempt  to  quantify  the  information  loss  going  from  15 
to  12  and  from  12  to  9  samples  to  insure  that  the  reduction  in  samples  is  truly  cost 
efficient. 
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The  initial  design  points  or  Starting  Values  (SV)  considered  are  V50  -  3cr,  V50  -  lcr. 
V50.  and  V50  4-  3cr.  V50  was  chosen  as  the  idea!  starting  value,,  and  V50  -  lcr  is 
considered  as  being  only  slightly  off  the  ideal  starting  point.  For  the  Cauchy 
distribution  any  quantity  defined  in  terms  of  a  will  be  calculated  with  (7=1.  The 
symmetric  response  functions  being  used  will  only  consider  error  to  one  side  of  the  true 
median.  For  asymmetric  distributions  it  is  necessary  to  consider  error  on  both  sides  of 
the  true  median. 

Gates  for  the  Langlie  will  be  SY  ±  lcr.  SV  ±  3 cr,  and  SV  ±  5 a.  They  will  be 
denoted  G\V(1).  G\V(3),  and  GW(5),  respectively.  This  should  give  us  a  Aide  enough 
range  so  that  an  indication  of  poor  gate  setting,  if  it  has  an  effect,  will  be  noticeable  in 
the  results.  In  practice,  setting  of  gates  is  done  using  the  information  that  the 
experimenter  supplies.  This  information  affects  all  the  designs  in  the  study.  In  section 
4.2  we  relate  gate  width  of  the  Langlie,  to  the  truncation  constant  c  shared  by  the  other 
designs. 

Three  types  of  noise  were  used:  none,  symmetric,  asymmetric.  Most  research 
concerns  itself  with  the  ideal  'no  noise'  situation.  We  also  want  to  include  this  common 
situation  so  that  our  results  may  have  application  outside  local  testing.  The  noise 
associated  with  firing  velocity  is  thought  to  be  symmetric  and  normal-like  in  behavior 
based  on  local  experience.  Hence  when  considering  symmetric  noise  we  will  use  a 
normal  density,  with  mean  0  and  standard  deviation  an.  We  can  also  imagine  situations 
in  which  noise  might  take  on  an  asymmetric  shape.  We  chose  to  use  an  exponential 
density  with  median  0  and  standard  deviation  <rn  to  represent  this  type  of  noise.  Choice 
of  the  value  of  an  for  each  of  the  noise  distributions  was  made  as  follows. 

Ten  sets  of  rec°nt  penetrator-against-plate  testing  was  analyzed  using  maximum 
likelihood  estimation  with  an  assumed  normal  response  function.  In  each  case,  the  NILE 
of  the  median  was  within  the  set  of  data  and  was  considered  good.  In  addition,  the 
MLE  of  the  response  distribution  standard  deviation  a  was  calculated  and 
approximately  corrected  for  bias.  Langlie  (1962).  The  sample  standard  deviation  <rd  of 
the  difference  between  nominal  and  actual  firing  velocity  was  computed.  In  the 
simulation  it  is  desirable  to  relate  the  noise  standard  error  to  the  populations  standard 
error.  The  pooled  results  of  the  ten  studies  indicated  that  a  representative  value  for  <7n 
is  .15(7. 

2.  Controls 


In  determining  the  number  of  iterations  to  be  used  for  each  set  of  conditions,  we 
did  some  preliminary  simulation  using  the  Langlie  design.  We  successively  tried  100  to 
1500  iterations  using  this  algorithm  with  12  design  observations  per  trial.  We  examined 
changes  in  maximum  likelihood  and  AVR  estimates  of  the  median  as  well  as  in  the  V50’s 
root  mean  square  error  >/MSE.  The  \/MSE  did  not  really  change  much  after  200  but  the 
mean  of  the  empirical  estimate  density  for  both  AVR  and  NMLE  stabilized  at  700 
observations.  Thus  we  chose  700  as  the  number  of  iterations  to  be  used. 
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The  random  number  sequence  will  be  used  in  the  following  manner.  Each  of  the 
five  designs  use  the  exact  same  random  number  sequence  for  a  given  cross  of 
factors.  All  three  estimators  within  each  design  and  for  a  given  cross  of  factors  are 
based  on  design  points  resulting  from  this  common  sequence  of  random  numbers.  This 
arrangement  nests  estimator  within  design;  estimators  will  be  compared  only  within  a 
given  design.  However  combinations  of  different  designs  and  estimators  can  be 
compared.  Discussion  of  the  random  number  sequence  is  included  in  Appendix  A. 

Based  on  Wu's  1985  results  and  those  of  preliminary  simulations  of  the  EQRC- 
DRM  and  EQRC-ARM  performed  by  us,  it  seemed  that  some  truncation  was  desirable 
for  EQRC  and  ARM.  However,  we  wanted  to  avoid  excessive  truncation  which  would 
limit  the  potential  of  the  design  to  make  large  moves  when  appropriate.  The  question 
we  battled  with  was  how  much  truncation  should  we  expect  to  have  if  the  designs  are 
operating  at  their  best.  Early  study  showed  that  a  truncation  constant  of  6cr.  more 
severe  than  any  that  Wu  considered,  really  performed  very  similar  to  a  constant  of  20cr 
in  terms  of  vAlSE  of  V50  for  the  cases  considered.  So  we  felt  it  reasonable  to  consider 
some  small  constants  for  this  small  sample  situation. 

Achieving  comparability  among  the  five  designs  requires  an  equivalence  between 
the  Gate  Width  and  the  truncation  constant,  c.  Recall  that  Starting  \alue  and  Gate 
Width  or  truncation  constant  comprise  the  usable  prior  information.  Starting  \  allies 
present  no  problem  for  comparability,  but  Gate  Width  and  truncation  do.  Larger  values 
for  each  encourage  a  wider  range  of  sampling,  and  smaller  values,  a  more  narrow 
range.  A  restriction  in  sampling  range  is  an  advantage  provided  the  range  covers  the 
true  median.  For  all  but  the  Langlie  a  maximum  move  of  c/2( n-k-h  1 )  may  be  made  to 
collect  the  (n+l)st  sample,  assuming  the  first  reversal  occurred  at  trial  k  (c/2  with  no 
reversal).  The  Langlie  procedure,  with  out  a  monotone  nonincreasing  step  size,  can 
move  at  most  GW/2  at  any  step.  The  level  of  restriction  for  each  will  be,  in  a  sense, 
the  same  if  we  set  c=GW. 

This  restrictive  situation  has  the  following  implications.  First,  superior 
performance  among  any  of  the  RM  type  procedures  can  be  directly  attributed  to  the 
procedure’s  superior  design  point  selection  and  not  to  varying  freedom  in  design  point 
range.  Second,  well  chosen  gate  widths  should  give  a  better  idea  of  good  and  bad 
truncation  constants  for  all  RM  procedures.  Third,  a  practical  comparison  among  the 
designs  is  facilitated  by  the  truncation  relationship.  Note  that  some  gates  chosen  as 
factor  levels  act  to  limit  the  flexibility  of  a  design.  How  much  limitation  they  impose 
should  also  be  related  to  Starting  Value.  The  truncation  constant  is  limiting  only  if  it 
causes  the  design  to  use  the  truncation  constant  in  calculation  of  the  next  design  point 
an  excessive  number  of  times.  Based  on  the  early  simulations,  1  or  2  truncations  for  a 
sample  size  of  15  seemed  reasonable.  Truncation  will  be  taken  into  account  in  the 
analysis  if  some  test  conditions  cause  truncations  to  be  excessive. 


i: 


3.  Measured  Variables 


The  response  for  this  experiment  is  taken  to  be  the  empirical  distribution  of  V50  as 

characterized  bv  its  mean,  and  its  root  mean  square  error. 

1  700  . 

SQRT(  — V(V50_\.0)-).  The  empirical  density  of  the  estimators  will  also  be 

examined.  In  the  following  this  density  will  often  be  denoted  "empirical  estimate 
density.”  Analysis  will  concentrate  on  the  \/MfeE.  We  also  measured  some  counters 
indicating  how  much  truncation  (gate  calling  for  Langlie)  was  taking  place  during  the 
simulation.  In  addition  we  recorded  the  existence  or  nonexistence  of  unique  maximum 
likelihood  estimates.  .411  of  these  measured  variables  will  figure  into  the  observations 
made. 

4.  Design  Configuration 

The  design  layout  for  this  study  can  be  seen  in  Table  1,  and  the  method  of  data 
collection  is  outlined  in  Figure  2.  The  reasons  for  our  design  configuration  are  as 
follows.  The  assumption  of  normality  is  commonly  made  for  our  day  to  day  data.  In 
addition  the  parametric  procedures  discussed  here  are  based  on  the  normal  and  logistic 
distributions.  Hence  most  of  the  observations  were  taken  from  this  distribution.  The 
asymmetric  noise  considered  is  really  not  thought  to  apply  to  our  specific  problem  so  we 
looked  at  it  only  in  conjunction  with  the  normal  distribution.  We  used  the  two 
common  noise  conditions  with  each  of  the  other  response  distributions  considered.  .4s 
the  normal,  uniform  and  Cauchy  densities  are  symmetric,  we  felt  it  necessary  only  to 
consider  Starting  Values  on  the  lower  side  of  the  median.  Gate  Width  and  Starting 
Value  combinations  which  do  not  cover  (within  the  gates)  V60  are  predictably  bad 
situations.  We  looked  at  three  such  combinations,  only  for  the  normal  response 
function,  to  see  how  much  worse  those  cases  are.  In  the  case  of  the  exponential  and 
empirical  response  functions,  we  chose  to  consider  five  Starting  Value  and  Gate  Width 
combinations.  Three  start  the  design  at  the  median  with  varying  Gate  Widths.  The 
remaining  two  will  compare  starting  the  design  toward  the  short  tail  of  the  distribution 
as  opposed  to  the  long  tail.  With  this  design  configuration  we  will  be  able  to  look  at  all 
of  the  items  of  interest  stated  previously. 


V.  ANALYSIS 


I.  Observation  Methods 


Because  of  difficulties  associated  with  applying  formal  statistical  tests  to  this  type 
of  data,  analysis  is  confined  to  graphical  and  table  summarization  of  these  Monte  Carlo 
results.  Each  effect  is  quantified  using  these  summarizations.  This  amounts  to  a  case 
by  case  comparison  of  values  in  terms  of  \/MSE,  V50,  and  the  empirical  density  of  the 
estimator  with  more  emphasis  being  given  to  \/MSE.  For  example,  consider  Figure  3 
containing  values  of  \/MSE  for  each  of  two  estimation  procedures  over  12  practical 
situations.  The  factor  of  interest  discussed  here  will  be  noise.  We  consider  method  B 
and  A  about  the  same  for  zero  noise,  but  B  is  generally  superior  to  A  over  the  range  of 
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TABLE  1. 


DESIGN  MATRIX 


— 

RESPONSE 

CVR'.'E 

S.AMPLE 

SIZE 

ZERO 

NOISE 

SV  t  GW 

NORMAL 

NOISE 

SV  &  GW 

ASYMMETRIC 

NOISE 

SV  &  GW 

A  so  ~  3(7)  i  Iff 

ft 

> 

(Vw  -  3cr)  ± 

«» 

n 

A  so  ~  3<7 )  ±  5(7 

ft 

ft 

A'so  -  la)  ±  Iff 

tt 

" 

NORMAL 

9. 12. 15 

A'so  -  Iff)  ±  3(7 

* 

ft 

A  so  -  Iff)  i  5<7 

ft 

n 

A'so)  ±  Iff 

ft 

n 

A'so)  ±  3(7 

ft 

ft 

(V*o)  ±  5(7 

tt 

(\  —  1(7 )  ±  3(7 

” 

UNIFORM 

15 

( Vi0  -  Iff)  ±  off 

t« 

NONE 

(V6o)  ±  Iff 

ft 

A  so)  i  3(7 

ft 

A  so)  i  5(7 

ft 

(Vw  -  3)  ±  5 

tt 

A'so  -  1)  ±  3 

n 

CAUCHY 

15 

(VM  -  1)  ±  5 

tt 

NONE 

(V60)  ±  1 

ft 

A’so)  ±  3 

tt 

A'so)  ±  5 

ft 

60  —  3<7 )  i 

ft 

(V*o)  ±  1(7 

ft 

EXPONENTIAL 

15 

(Vj  ±  3(7 

ft 

NONE 

A&o)  ±  5(7 

ft 

(Vyj  +  3(7)  ±  5(7 

ft 

A  so  _  3ff)  ±  5<7 

ft 

A'so)  ±  l<r 

ft 

EMPIRICAL 

15 

A’so)  ±  3(7 

ft 

NONE 

(V»)  ±  5(7 

ft 

(Vjq  •+•  3(7)  i  3(7 

it 
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Start 

[  «£ - 

Initialize  random  number  seed 

Choose  a  design 

< - 

Choose  other  factor  levels 


Compute  \  50,  v/MSE,  average  truncation,  #  MLE  exist 

I 

All  factor  level  combinations?  - No  - >■ 

Yes 

I. 

All  designs? - No - > 

Yes 
Stop 

Figure  2.  Flowchart  of  observation  collection 
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normal  and  asymmetric  noise  cases.  Further  B  seems  relatively  insensitive  tc  either 
type  noise,  whereas  method  A  generally  has  a  higher  vMSE  with  both  normal  and 
asymmetric  noise.  Case  2  is  different  in  that  it  always  contains  the  worst  showing  for 
method  B.  Other  factor  levels  would  be  examined  to  help  explain  this. 

Noise 


Equivalent  Case  Sets 

Figure  3.  Comparison  of  two  estimation  procedures. 


This  type  of  graphical  study  is  fairly  representative  of  our  approach  to  making 
observations.  Above,  when  directly  contrasting  the  two  methods,  we  are  looking  at  two 
things  primarily:  the  number  of  times  that  B  was  less  than  A,  the  magnitude  of  the 
difference  between  B  and  A.  The  smallest  difference  that  we  are  going  to  consider  is 
.Oder.  The  reason  for  this  is  that  the  primary  function  of  this  study  is  to  make  a 
recommendation  as  to  design  and  estimation  for  our  weapons  environment.  The 
smallest  noise  situations  that  we  observed  when  characterizing  an  had  standard 
deviations  of  approximately  ,04<7.  In  our  environment,  to  favor  method  B's  estimate  of 
V50  over  method  A's  when  A  and  B  differ  by  less  then  .04cr  is  not  wise  as  we  would  be 
favoring  one  velocity  estimate  over  another  when  their  difference  is  within  the  noise  of 
our  Bring  capability. 

One  last  note  concerning  our  analysis  involves  the  handling  of  the  empirical 
response  function.  Similar  to  an  approach  in  modeling,  where  one  saves  a  portion  of  the 
data  to  be  used  as  a  check  against  the  model,  we  have  chosen  to  use  results  from  the 
empirical  response  function  as  a  check  of  observations  made  regarding  the  other  three 
response  functions. 

2.  Estimate  Existence  and  Truncation 

Prior  to  making  observations  regarding  the  factors  in  the  experiment,  we  will  make 
some  statements  concerning  the  measured  variables  other  than  the  response.  For  G\V(1) 
over  all  SV’s,  the  maximum  likelihood  estimates  did  not  exist  uniquely  approximately 
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3 °c  to  loFc  of  the  time;  for  the  Langlie.  in  the  eases  where  ihe  gates  did  cover  V 50.  they 
failed  to  exist  uniquely  approximately  3*7-  to  24*7-  of  the  time.  Of  course,  the  exception 
is  that  when  the  gates  did  not  cover  V50  for  the  Langlie,  the  NILE  failed  to  exist 
uniquely  almost  all  of  the  time.  This  problem  of  unique  estimates  was  particularly  bad 
for  SV(-3),  GVV(l)  which  generally  yields  the  higher  percentage.  With  the  wider  gate 
widths,  (GW(3),  and  GW(5),  regardless  of  SV).  the  MLE  exists  about  95°?  of  the 
time.  It  is  interesting  that  for  the  normal  response  function,  additional  samples  only 
sightly  increase  the  chance  of  MLE  unique  existence,  thus  reminding  us  of  the 
importance  of  the  beginning  samples. 

As  expected.  SV  and  GW  were  the  determining  factors  in  controlling  the  amount  of 
truncation  which  took  place.  When  the  gate  width  was  just  1,  and  the  starting  value 
was  (-3),  the  number  of  EQRC  truncations  were  restrictive,  sometimes  truncating  as 
much  as  607  of  the  time.  In  fact,  the  number  of  truncations  for  GW(1)  and  both 
SY(-l)  and  SY(0)  is  still  fairly  high.  For  GW(3)  and  GW(5)  the  situation  was  much 
better.  In  the  case  of  EQRC-DRM  and  EQRC-ARM  the  average  amount  of  truncation 
taking  place  per  iteration  for  sample  sizes  9.  12.  and  15  was  less  than  1,  less  than  2.  and 
approximately  2  respectively.  Of  the  two.  GW(5)  generally  causes  slightly  less 
truncation  than  GW(3). 

In  the  case  of  .ARM,  it  is  also  true  that  GW(1)  has  a  severe  effect  on  the  number  of 
truncations  made  and  that  GW(3)  and  GW(5)  are  much  less  restrictive.  By  comparing 
the  number  of  truncations  of  ARM  to  the  number  of  truncations  from  the  ARM  portion 
of  EQRC-.ARM.  we  determined  that  the  greatest  number  of  truncations  are  occurring 
fairly  late  in  the  data  collection  for  .ARM. 

In  the  case  of  the  Langlie  design.  SY(0)  regardless  of  GW,  cau^s  the  average 
number  of  calls  to  either  gate  to  be  between  1  and  2  for  all  sample  sizes.  For  other 
SY’s,  the  number  of  calls  to  the  gates  rise  considerably.  This  situation  is  at  its  worst 
when  the  gates  fail  to  cover  the  median. 

In  summary,  when  comparing  designs,  it  will  be  necessary  to  examine  most  closely 
those  cases  in  which  the  number  of  truncations  are  reasonable.  For  this  reason, 
attention  will  be  given  to  GW(3)  and  special  attention  to  GW(5)  since  all  the  designs 
seemed  to  be  fairly  free  to  move  in  those  environments.  The  extent  to  which  the 
maximum  likelihood  estimation  can  be  relied  upon  in  combination  with  the  design  will 
also  be  considered. 

3.  Sample  Size 

For  the  normal  response  function  and  all  estimation  techniques  and  designs,  \/MSE 
decreases  with  increasing  Sample  Size.  The  precision  gain  of  6  additional  samples  in 
terms  of  the  v/MSE  associated  with  NMLE  averages  .13<r  for  all  the  RM  type  designs 
and  .10<r  for  the  Langlie  strategy.  See  Figure  4.  In  the  figures  that  follow,  ’’case'1  is 
meant  to  mean  a  SV  and  GW  combination.  The  precision  gain  of  6  samples  for  AVR’s 
\/MSE  averages  about  .  11<7  for  RM  type  designs  and  .08cr  for  Langlie.  Precision  gain  for 
Next  Stress  ranges  between  .06(7  and  . I2cr.  For  the  normal  response  function  and  all 
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estimation  techniques  for  the  R.M  type  designs,  increased  sample  size  accentuates  the 
effect  of  asymmetric  noise.  This  effect,  a  biasing  of  \50.  will  be  discussed  later. 

Sample  Size  (SS) 


Figure  4.  Effect  of  sample  size  on  v^MSE. 


4.  Noise 


In  general  for  NMLE.  A\R,  and  Next  Stress,  zero  noise  and  normal  noise  seem  to 
produce  the  same  range  of  estimates  for  Vjq,  whereas  asymmetric  noise  will  drive  the 
estimates  toward  the  upper  tail  of  the  response  function.  Examples  are  given  in  Figures 
5  and  6.  This  observation  is  consistent  with  the  appearance  of  the  empirical  densities 
for  all  the  estimation  techniques.  There,  asymmetric  noise  causes  a  slight  skewness 
toward  the  higher  velocities,  particularly  for  narrow  Gate  Width  cases.  In  addition, 
asymmetric  noise  may  shift  the  median  away  from  the  known  value.  Normal  and  zero 
noise  situations  leave  an  empirical  estimate  density  which  is  symmetric  and  centered 
about  zero,  the  known  V50.  The  exception  to  this  observation  is  with  Langlie  which 
seems  relatively  insensitive  to  noise  except  for  the  following  isolated  cases.  The  first 
situation  is  that  V50  for  SV(-3)  and  GW(5)  is  higher  for  asymmetric  noise  than  for  zero 
or  normal  noise.  The  second  is  that  normal  noise  acts  to  drive  upward  the  Next  Stress 
estimate  of  V50  for  an  exponential  response  function.  The  effects  of  noise  on  \50  are 
more  noticeable  for  the  DRM  and  ARM  designs  regardless  of  estimation  technique.  As 
stated  previously,  larger  sample  sizes  make  the  effects  of  noise  on  V50  more  noticeable. 


18 


m  -o  (o  vl>  .  .....  pj  r*l 


SYMBOL  COUNT  MEAN  ST. DEV. 

X  700  0.192  0.199 

EACH  SYMBOL  REPRESENTS  1  OBSERVATIONS 

R7AL 

5  1C  15  20  25  30  35  40  45  50  55  60  65  7. 

+ - + - 4 - + - + - + - + - + - + - + - _ - * - 

5  + 

4  + 

3  + 

2  + 

1  + 

+ 

9  + 

8  + 

7  + 

6  + 

5  +  X 

4  + 

3  + 

2  + 

1  +X 

+  X 
+XX 
+  XXXX 

+ X  X  X  X  X  X  X  X  X  X  X 
+XXXXXXXXXXXXXXXXXXXXX 
+XXXXXXXXXXXXXXXXXXXXXXXXXX 
. 4  +XXXXXXXXXXXXXXXXXXXXXX 

*  - . 3  +XXXXXXXXXXXXXXXXXXXXXXXXXXXX 

*  - . 2  +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 

* - . 1  +XXXXXXXXXXXXXXXXXXXXXXXXXXXX 

*-134E-9  +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 

*. 1  +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 

*  .  2  +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 

*.3  +xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx::h>: 

* . 4  +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 

* . 5  +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 

*  .  6  +XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 

* . 7  +XXXXXXXXXXXXXXXXXXXXXX 

* . S  +XXXXXXXXXXXXXXXXXXXX 

* . 9  +XXXXXXXXXXXXXXXXXXXXXXXX 

*:  +xxxxxxxxxxxxxxxxxxxx 

*1.1  +XXXXXXXXXXXXXXXX 

*1.2  +XXXXXXXXXXX 

*1.3  +XXXXXXXXX 

*1.4  +XXXX 

*1.5  + 

*1.6  +X 

*1.7  + 

*1.8  +■ 

*1.9  + 

*2  +XX 

*2.1  + 

*2.2  + 

*2.3  + 

*2.4  + 

*2.5  + 

*LAST  + 

+ - + - + - + - + - + - + - + - + - + - + - + - + - V - 

5  10  15  20  25  30  35  40  45  50  55  60  65  70 

Figure  5.  Effect  of  asymmetric  noise  on  Next  Stress 


19 


Noise 


zero 


c 

»  o 

<> 


0.2 


O.i 


0.0 -j 


— 0.1  — { 


— 0-2 


rr.ecn  =  —.02 


normal 


rrecn  =  —.02 


asymmetric 


no rmct  response 
somcie  sire  15 
AVR 

ECRC— DRW 
mecn  —  .06 


“  I  t  I  I  I  I  i  I  T 

123*56763 


"7  I  i  !  t  t  I  i  i 

123456789 


t — i — i — i — r 

2  3  4  5  6 


7  6  9 


Eauivalent  Case  Sets 


Figure  6.  Effect  of  noise  on  \\ 


SO- 


The  estimation  technique  seems  to  interact  with  noise  for  \/MSE.  For  each 
collection  procedure,  the  range  of  NMLE  and  A\"R  estimates'  vMSE  is  about  the 
over  all  noise  within  a  response  function  and  sample  size.  See  Figure  7. 
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Figure  7.  Effect  of  noise  on  v^MSE  for  AVR. 
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However,  the  \/MSE  associated  with  Next  Stress  inflates  some  for  norma;  noise  and  still 
more  for  asymmetric  noise.  This  trend  is  more  noticeable  for  increased  samples.  It  is 
also  more  evident  for  DRM  and  ARM  than  for  EORC.  This  is  an  important  observation 
since  Next  Stress  is  :^e  intended  estimation  procedure  for  the  RM  designs.  See  Figure 
8.  For  Langlie,  stress  varies  so  much  that  such  a  claim  would  be  difficult  to  make. 
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Figure  8.  Effect  of  noise  on  v/MSE  for  Next  Stress. 

5.  Starting  Value  and  Cate  Width 

Starting  Value  and  Gate  Width  interacted  with  noise,  sample  size,  design,  and 
estimation.  As  one  might  expect  general  rules  are  difficult  to  find.  The  following  are 
some  of  the  notions  inferred  from  observing  the  results  obtained  by  varying  SY  and 
GW.  . 


GW's  causing  severe  truncation  usually  limit  the  designs’  ability  to  rebound  from  a 
poor  decision  in  SV  or  a  poor  decision  during  the  data  collection  process  thus  causing 
higher  \/MSE’s.  An  exception  to  this  is  SV(0)  and  GW(1)  where  limited  movement 
forces  the  design  to  collect  data  close  to  the  median. 

Among  SV’s  and  GW’s  promoting  light  truncation,  GW(5)  seem  to  produce 
reasonable  results  relative  to  the  more  narrow  gates.  For  sample  size  9,  GW(1)  and 
GW(3)  were  usually  better  performers  than  GW(5).  However,  for  15  samples  all  the 
GW's  performed  similarly.  For  the  exponential  response  function  there  was  no  real 
difference  between  SV(-3)  and  SY(3),  with  GW(5).  This  was  probably  due  to  GW(5) 
rather  than  the  designs’  indifference  to  SV  for  an  asymmetric  distribution. 
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SV  and  GW  combinations  such  that  the  median  was  at  a  gate  or  outside  both  gates 
for  the  Langiie  were  predictable.  If  the  median  was  at  the  gate  the  algorithm  would 
begin  to  converge  toward  the  gate.  Thus,  the  NMLE  procedure  would  appear  to  be 
very  good  in  this  situation.  However,  if  the  median  was  outside  of  the  gates,  it  is 
impossible  for  a  reasonable  estimate  of  the  median  to  be  made. 

The  overriding  element  in  all  of  these  observations  is  simply,  altering  the  SY  and 
GW  will  affect  the  designs’  ability  to  gather  information  near  the  median.  The  design 
needs  to  be  able  to  rebound  from  an  incorrect  decision  (different  than  "ideal" ).  For 
instance,  if  we  consider  SV('-3)  and  GW(2),  it  is  quite  possible  to  be  sampling  at 
V50  4-  bo  on  the  third  sample.  These  kind  of  wrong  decisions  are  very  detrimental  in 
small  sample  problems. 

6.  Response  F unction 

For  performance  over  different  response  functions,  we  only  considered  a  sample  siz^ 
of  15.  In  general,  over  all  estimation  techniques  and  designs,  normal,  uniform  and 
Cauchy  response  functions  yield  V^o^O;  whereas  for  exponential.  V50^.12  for  R\1 
designs,  and  .16  for  Langiie.  This  is  true  of  both  normal  and  zero  noise  situations.  One 
exception  seen  is  (  V50  -  3<7  )  ±  5<r  where  occasional  outliers  for  the  Cauchy  response 
function  lowers  the  average  estimate.  See  Figure  9. 
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Figure  9.  V50  for  four  response  curves  with  zero  and  normal  noise. 
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For  the  Cauchy  distribution,  although  still  centered  at  zero,  we  begin  to  pick  up 
outliers  in  the  empirical  estimate  density.  Outliers  as  far  out  as  2.5  from  zero  as 
compared  to  1.4  for  norma!  were  observed.  It  is  easy  to  imagine  why  this 
occurred.  With  the  Cauchy,  when  sampling  in  the  tail  there  is  a  greater  probability  of 
an  incorrect  decision  thus  sending  the  sequential  algorithm  in  the  wrong  direction.  This 
is  an  important  consideration  since  in  practice  the  underlying  response  distribution  is 
unknown.  On  a  small  sample  basis,  what  might  appear  to  be  normal-like  even  to  the 
point  of  matching  quartiles,  may  in  fact  be  Cauchy,  the  heavy  tails  of  which  could  cause 
severe  estimation  problems  on  occasion. 

For  NMLE  and  AVR  estimation  for  each  design,  the  \/MSE  for  the  uniform  is 
slightly  higher  (around  .06>t)  than  \/MSE  for  the  normal  over  most  cases.  This  also 
seems  to  be  true  occasionally  with  RM  designs  and  Next  Stress  but  there  is  too  much 
variability  to  ascertain  a  general  rule.  This  is  the  same  observation  made  by  Davis 
(1969).  Even  with  the  outliers,  the  Cauchy  response  generally  yields  vMSE's  iower  than 
those  of  the  normal.  See  Figure  10.  This  is  a  function  of  the  amount  of  probability 
mass  immediately  around  the  median.  This  point  will  be  addressed  in  a  following 
plot.  Although  V*5o  for  the  exponential  response  function  is  higher,  vNISE  is  generally 
lower  (around  .  10cr)  than  for  the  uniform  response  functions. 

Response  Curve 


Equivalent  Case  Sets 

Figure  10.  vMSE  for  four  response  curves  and  zero  and  normal  noise. 

This  is  true  of  all  designs  and  estimators  except  the  Langlie.  For  the  Langlie  strategy 
the  difference  is  more  like  .05<r.  The  primary  reason  for  these  response  distribution 
differences  is  discussed  below. 


Figure  11  shows  the  normal,  exponential.  Cauchy  and  uniform  densities  used  in  this 
analysis.  Comparative  evaluation  of  the  distribution  functions  at  particular  points 
about  the  median  will  shed  some  light  on  the  above  paragraph.  Banerjee  (1980)  showed 
that  in  the  case  of  maximum  likelihood  estimation  of  the  median  of  the  response 
function,  a  concentration  of  observations  about  the  median  would  allow  for  an  efficient 
estimate.  It  then  makes  sense  that  the  order  of  v'AlSE  from  lowest  to  highest  was 
Cauchy,  exponential,  normal,  and  uniform  since  the  concentration  of  probability  mass 
about  the  median  also  shares  that  increasing  order.  It  is  also  makes  sense  to  expect  an 
estimate  biased  upwards  for  the  exponential  response  since  the  estimates  will  be  either 
near  the  median  or  sometimes  much  higher.  We  examined  the  empirical  estimate 
densities  for  the  various  response  functions  also.  Normal  and  Cauchy  response  functions 
yielded  fairly  peaked  and  fairly  symmetric  empirical  estimate  densities,  whereas  the 
uniform  yielded  a  much  flatter  density  but  also  symmetric.  The  exponential  response 
caused  all  empirical  estimate  densities  to  be  very  peaked  and  skewed  toward  higher 
estimates  of  V50.  Normal  and  Cauchy  examples  for  conditions  DRM,  N\ILE,  zero  noise, 
SS(  15),  and  (  V50  -  3cr )  ±  5 a  are  given  in  Figures  12  and  13.  Uniform  and  exponential 
examples  for  the  same  set  of  test  conditions  are  given  in  Figures  14  and  15. 
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Figure  12.  Empirical  estimate  density  for  Cauchy  response 
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Figure  15.  Empirical  estimate  density  for  normal  response 
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Figure  14.  Empirical  estimate  density  for  uniform  response 
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Figure  15.  Empirical  estimate  density  for  exponential  response 
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7.  Estimation 

One  important  observation  is  that  AVR  and  NMLE  perform  virtually  the 
same.  There  seems  to  be  a  slight  tendency  of  NMLE  to  improve  relative  to  AVR  with 
increased  sample  size  and  NMLE's  estimate  of  V50  is  not  affected  as  much  as  AVR’s  for 
the  exponential  response  function.  Furthermore,  because  of  the  way  AVR  is  computed, 
the  heavy  tails  of  the  Cauchy  could  cause  a  very  low  response  or  a  very  high 
nonresponse  to  be  figured  into  the  average  and  possibly  cause  an  outlier.  Other  than 
for  those  three  situations,  the  small  sample  nature  of  this  study  makes  them  about  the 
same.  Remember,  NMLE’s  backup  procedures  include  AVR.  Still,  even  in  the  cases  in 
which  NMLE  exists  about  QS0©  of  the  time,  AVR  performs  the  same.  See  Figure  16. 

Noise 


Equivalent  Case  Sets 


Figure  16.  AVR  and  NMLE  estimation  subject  to  noise. 


Only  under  zero  noise  do  Robbins-Monro  Next  Stress  estimates _ perform 

consistently  as  well  as  AVR.  and  NMLE.  Asymmetric  noise  causes  V*50  and  >/MSE  to  be 
poorer  (higher)  for  Next  Stress  relative  to  NMLE  and  AVR.  This  is  also  true  of  normal 
noise  when  considering  only  v/MSE.  See  Figure  17.  Empirical  densities  show  this  by 
yielding  a  flatter  density  for  Next  Stress  than  for  AVR  or  NMLE.  This  effect  of 
asymmetric  noise  is  more  evident  with  larger  samples  and  applies  only  to  RM 
designs.  The  reason  for  large  sample  effect  is  that  there  are  simply  more  opportunities 
for  the  noise  to  lead  the  design  away  from  the  median  thus  causing  greater 
variability.  For  DRM  and  ARM  we  would  expect  this  of  small  sample  problems  since 
noise  introduces  a  greater  chance  for  a  wrong  decision.  For  EQRC-DRM  and  EQRC- 
ARM  and  heavy  truncation,  the  Next  Stress  (a  logit  MLE)  may  not  be  able  to  get  back 
to  the  median.  However,  even  in  light  truncation,  the  LMLE  Next  Stress  performs 
slightly  worse  overall  than  the  NMLE  acting  on  the  same  design  points. 
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Figure  17.  All  estimation  subject  to  noise. 


There  are  at  least  two  possibilities  why  this  occurred  One  is  that  for  these 
restrictive  samples,  the  LMLE  is  not  being  given  enough  opportunities  to  zero  in  on  the 
median  because  of  the  base  design.  Another  is  that  with  small  samples,  possibly  the 
approximation  to  logit  maximum  likelihood  estimation  is  affected  by  use  of  design  points 
far  from  the  median. 

Over  the  entire  scenario  of  cases,  even  for  nonnormal  response  functions,  NMLE 
performs  as  well  or  better  in  terms  of  \/MSE  than  AVR  or  Next  Stress.  Only  for  the 
exponential  or  Cauchy  response  would  we  much  prefer  NMLE  over  AVR. 

8.  Design 

The  designs  are  compared  twice.  First,  each  design  with  its  intended  estimation 
procedure  and  second,  each  with  its  best  estimation  procedure.  Next  Stress  is  intended 
for  the  RM  designs  and  NMLE  for  the  Langlie.  NMLE  was  the  best  estimation 
procedure  for  all  designs.  The  ranking  of  designs  with  their  intended  estimation  for 
light  truncation  where  procedures  were  not  severely  restricted  in  design  point  selection 
by  the  truncation  constant, 

Langlie  >  EQRC-DRM  =  EQRC-ARM  >  ARM  >  DRM, 

where  >  indicates  "better  than."  For  heavy  truncation  where  procedures  were 
restricted, 


EQRC-DRM  =  EQRC-ARM  >  ARM  =  DRM  >  Langlie. 

Langlie  completely  reverses  its  position  in  our  judgement  because  of  its  inability  to 
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estimate  in  some  heavy  truncation  situations.  In  general  Langlie  performs  better  for  th<> 
uniform  response  than  the  others  and  RM  designs  are  hurt  by  noise  environments 

The  final  design  comparisons  takes  the  best  estimator’s  results  associated  with  each 
design  and  compares  them  among  all  designs.  It  turned  out  that  the  NMLE  yielded  the 
best  estimate,  given  the  data  gathered  by  each  design.  In  this  situation  for  light 
truncation 


I)RM  =  ARM  =  EQRC-ARM  =  EQRC-DRM  =  Langlie. 


In  the  presence  of  heavy  truncation. 


DRM  =  .ARM  =  EQRC-ARM  =  EQRC-DRM  >  Langlie. 

since  Langlie  is  again  unable  to  estimate  in  some  heavy  truncation  situations.  Shown  in 
Figure  18  is  the  \/MSE  of  each  design  using  NMLE  over  three  GW  and  SY(0) 
combinations.  These  combinations  included  some  light  truncation  cases  shared  by  all 
response  functions. 
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Figure  18.  Design  comparison. 


VI.  CONCLUSIONS 


As  reported  by  others,  we  found  that  poor  Starting  Values  and  Gate  Widths  cause 
problems  in  estimating  Y-0.  Their  primary  effects  are  to  limit  the  number  of  times  that 
the  maximum  likelihood  estimates  exist  uniquely  and  to  cause  truncation  in  design  point 
selection  to  be  severe.  Both  amount  to  restricted  gathering  of  information  near  the 
median,  causing  poorer  estimates.  Although  it  is  sensible  to  say  that  extremely  wide 
gates  will  waste  the  first  few  rounds  fired.  we  did  not  demonstrate  it  here  with  Gate 
Width  [±ha).  Apparently.  Gate  Width  (±5cr)  was  not  wide  enough  to  generally  cause 
wasted  rounds. 

The  response  distributions  we  sampled  from  did  make  a  different  in  maximum 
likelihood  and  average  estimation.  The  vNlbE  associated  with  each  estimate  is  giv“ii 
from  lowest  to  highest  by  the  ordering:  Cauchy  and  exponential  (about  the  same), 
normal,  and  uniform.  This  order  indicates  that  the  median  is  easier  to  estimate  when 
the  probability  mass  of  a  distribution  is  more  dense  around  the  median.  Furthermore, 
the  heavy  tails  of  the  Cauchy  caused  outliers  in  the  empirical  estimate 
density.  However,  a  Cauchy  distribution  and  a  normal  distribution  with  matching 
quartiies  are  not  easily  distinguishable  in  the  field  environment.  Experimenters  need  to 
be  aware  of  this  heavy  tail  situation  particularly  where,  on  the  basis  of  sketchy 
historical  data,  norm  •  lit y  is  assumed.  We  personally  have  seen  some  ballistic  data 
which  would  fall  into  this  heavy-tailed  distribution  category. 

There  is  a  noticeable  difference  in  the  precision  of  estimators  as  Sample  Size  is 
varied.  Changing  from  sample  size  9  to  sample  size  15  will  gain,  for  example.  ^  .13cx  in 
precision  of  the  WILE  normal  based  maximum  likelihood  estimate  with  the  average  of 
stress  levels  as  a  back-up  procedure.  In  fact,  since  the  WlSE  for  sample  size  9  is 
generally  about  .52.  this  change  amounts  to  approximately  a  25c7-  increase  in  precision 
of  the  estimate  from  sample  size  9  to  sample  size  15. 

Only  asymmetric  noise  has  a  great  impact  on  the  estimation  of  Y50.  Generali)  it 
acts  to  shift  the  mean  of  the  Y^0’s  upward  in  varying  amounts.  However  in  the  case  of 
XMLE  and  AYR.  WISE  is  unaffected  by  noise.  In  our  testing  environment  this  is  an 
important  quality. 

An  important  conclusion  regarding  estimation  is  that  AYR  and  WILE  are  the  best 
performers  on  the  whole  over  the  range  of  design  conditions  and  that  they  perform 
virtually  the  same  in  terms  of  WISE.  However,  for  zero  noise  situations  the  intended 
estimation  of  the  RM  type  designs,  Next  Stress,  does  as  well  as  AYR  and  NMLE  in 
terms  of  WISE.  Therefore,  if  there  is  zero  noise  in  the  testing  environment  and  the 
response  distribution  is  not  heavy  tailed,  there  is  no  particular  advantage  to  using  one 
over  the  other.  In  normal  or  asvmmetric  noise  environments  it  may  be  better  to  use 
WILE  or  A\"R  instead  of  Next  Stress. 

Let  us  consider  the  intended  estimation  of  the  five  designs.  We  conclude  that, 
among  the  RM-type  designs,  Wu's  EORC  generally  performs  as  good  or  better  than 
DRM  and  .ARM  over  the  design  scenario.  This  can  be  attributed  to  more  efficient 
design  point  selection  for  two  reasons.  The  first  is  that  among  the  four,  all  were 


restricted  to  the  same  maximum  step  size  at  each  stage  of  the  design.  Second,  the 
estimator  is  the  next  design  point  chosen.  The  fact  that  VVu's  procedure  makes  a 
parametric  assumption  as  part  of  the  design  point  selection  does  not  invalidate  this 
claim  since  even  in  the  case  of  the  nonnormal  response  functions  EQRC,  is  still 
superior.  Of  the  two,  EQRC-DRM  and  EQRC-ARM,  there  is  no  difference  in 
performance  so  we  would  choose  the  first  because  it  is  easier  to  carry  out. 

When  given  good  Starting  Values  and  Gate  Widths,  the  Langlie  with  intended 
estimation  NMLE  will  perform  only  slightly  better  than  EQRC-DRM’s  Next  Stress. 
However,  poor  Starting  Values  or  Gate  Widths  are  disastrous  for  the  Langlie.  Hence, 
unless  sure  of  covering  the  median  with  reasonably  spread  gates  (a  doubtful  situation 
many  times  in  practice)  it  is  better  to  go  with  the  more  flexible  EQRC-DRM. 

One  of  the  most  important  things  learned  from  examination  of  possible  estimators 
in  this  small  sample  environment  is  that  the  NMLE  and  A\~R  methods  of  estimation 
fare  best  even  when  the  response  function  is  nonnormal.  It  is  then  reasonable  to  apply 
either  of  these  two  methods  of  estimation  to  data  gathered  by  any  of  the  desians 
considered  provided  the  distribution  is  not  heavy  tailed.  With  NMLE  as  the  estimator, 
there  is  really  no  difference  among  the  five  designs  when  the  Starting  Value  and  Gate 
Widths  are  chosen  well.  This  claim  is  made  since  there  is  no  one  design  which  is 
consistently  better  than  the  others.  However,  if  Starting  Values  and  Gate  Widths  arc 
not  chosen  well  the  RM  designs  exceed  the  performance  of  the  Langlie. 

Therefore,  we  recommend  using  the  Delayed  Robbins-Monro  with  the  three  level 
estimation  technique  NMLE  as  outlined  in  the  second  chapter  since  it  is  the  easiest  to 
implement  among  the  four  RM  designs.  We  would  suggest  using  this  with  a  guesstimate 
of  V50  ±  3cr.  Its  primary  advantage  over  the  current  Langlie  procedure  is  its  ability  to 
sample  in  the  correct  region  despite  poor  initial  estimates.  A  second  advantage  is  that 
the  NMLE  exists  uniquely  slightly  more  often  for  DRM  than  for  Langlie. 
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APPENDIX  A 


MONTE  CARLO  METHODS 

Basic  to  all  simulation  studies  is  a  good  uniform  random  number  generator.  It  is 
really  up  to  each  individual  to  establi.h  what  good  is.  Their  definition  should  be 
dependent  on  the  problem  with  which  they  are  faced.  Different  problems  require 
emphasis  on  different  acceptance  criteria  usual  for  uniform  (0.1)  random  numbers.  We 
have  set  forth  here  what  we  consider  to  be  the  important  aspects  of  random  numbers  in 
our  study  and  how  well  the  random  numbers  measure  up  to  these  standards. 

First,  we  wish  our  random  numbers  to  pass  some  goodness  of  fit  criterion.  We 
chose  a  chi-square  test  with  bins  of  width  .025.  We  generated  500.000  random  numbers, 
stopping  to  calculate  the  chi-square  test  statistic  after  each  1000  numbers.  Testing  at 
the  q  =  .10  level,  only  7  times,  ranging  from  16.000  to  161.000  were  we  abie  to  reject 
the  uniform  hypothesis.  At  no  time  could  we  reject  at  the  .05  level.  Note  that  it  is  not 
true  that  we  should  have  necessarily  observed  50  and  25  rejections  for  the  .10  and  .05 
levels  respectively  since  we  are  not  talking  about  separate  random  samples  for  each  test 
in  this  situation.  (We  have  cumulative  samples.)  Hence,  based  on  goodness  of  fit  cri¬ 
teria.  we  are  pleased  with  the  performance  of  the  random  number  generator. 

Although  it  is  important  that  large  sample  results  be  good,  more  important  to  our 
study  is  the  small  sample  behavior  of  the  random  numbers.  .Along  these  lines,  there  is  a 

certain  number  of  runs  of  length  1.  length  2 . that  one  would  expect  to  observe  as 

well  as  a  certain  number  of  total  runs  that  one  would  expect.  These  expectations  are 
listed  by  Rubinstein  (1981).  We  computed  the  empirical  distribution  of  the  number  of 
runs  and  compared  it  with  the  theoretical  distribution.  We  found  no  evidence  to  suggest 
that  the  empirical  distribution  differed  significantly  from  the  theoretical  distribution. 

Of  course  no  random  number  generator  should  repeat  its  seed.  This  one  does  not 
repeat,  at  least  up  to  the  500,000  numbers  that  we  considered. 

Looking  at  random  numbers  would  not  be  complete  without  plots.  We  plotted 
empirical  densities,  bivariate  scatter  plots,  etc.,  and  found  no  patterns. 

Random  deviate  generation  was  performed  in  the  usual  manner.  The  uniform  (0,1) 
generation  is  handled  by  a  system  routine.  We  wished  to  generate  uniform,  normal,  and 
exponential  random  deviates  each  with  0  median  and  standard  deviation  1.  In  addi¬ 
tion,  a  Cauchy  distribution  was  considered  with  0  median  and  quartiles  matching  a  nor¬ 
mal  (0.1)  distribution.  Normal  (0,1)  deviate  generation  was  accomplished  with  the  Box- 
Muller  results.  The  uniform  (-v^ ,  \/3),  exponential  (1,  -ln2)  and  Cauchy  (0,  .675),  were 
generated  by  setting  their  respective  distribution  functions  equal  to  the  random  number, 
r.  and  then  solving  for  the  random  deviate  as  a  function  of  r.  In  the  case  of  the  empiri¬ 
cal  distribution,  we  took  79  data  points,  formed  an  empirical  distribution  using  the 
'method  of  reversals'  and  then  smoothed  it  with  a  polynomial  to  facilitate  simulation. 
We  drew  our  random  deviate  from  this  function. 
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Noise  generation  was  accomplished  similarly  to  the  above.  In  either  the  normal  or 
exponential  case  w'e  had  a  distribution  with  0  median  and  standard  deviation  .15  We 
simply  added  this  random  noise  deviate  to  the  design  point  under  consideration  before 
comparing  it  to  the  random  deviate. 
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APPENDIX  E 


DAT.', 

We  have  included  in  Tables  B2  -  B16  the  >/MSE  for  each  set  of  test  conditions. 
Each  table  gives  the  results  of  crossing  all  test  condition  factors  for  one  design  and  one 
estimation.  The  presentation  of  the  observations  corresponds  to  the  format  of  the 
design  matrix  repeated  from  the  fourth  chapter  and  given  here  in  Table  Bl.  For 
example,  in  Table  B2,  .595,  .480,  .424  correspond  to  sample  sizes  of  9.  12.  15 
respectively  for  a  normal  response  function,  zero  noise,  and  (V5o  -  3<r)  ±  la  as  the 
representation  of  the  experimenter's  guesstimate.  Also  in  Table  B2,  .464  corresponds  to 
a  sample  size  of  15  for  a  uniform  response  function,  normal  noise,  and  (V50  -  3er)  ±  5 er  as 
the  representation  of  the  of  the  experimenter’s  guesstimate. 
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TABLE  Bl. 


DESIGN  MATRIX 


RESPONSE 

CUR\E 

SAMPLE 

SIZE 

ZERO 

NOISE 

SV  &  GW 

NORMAL 

NOISE 

SV  &  GW 

D  60  -  3(7)  ±  1(7 

n 

(V  60  “  3(7 )  ±  3(7 

n 

( v  £0  -  3(7}  ±  5(7 

(Vjo  -  1<7)  ±  1(7 

b 

NORMAL 

9,12,15 

A  SO  ~  1(7)  ±  3(7 

n 

(Vjo  -  1(7)  ±  5(7 

n 

(\  50 )  ±  l*7 

" 

(A  So)  ±  3(7 

n 

(Vso)  ±  5(7 

b 

(Vjo  -  3(7)  ±  5(7 

- 

(\  £0  -  1(7)  ±  3(7 

* 

UNIFORM 

15 

(Vjo  -  1(7)  ±  5(7 

D  so)  ±  1^ 

(!■  so)  ±  3(7 

(Vjo)  ±  5(7 

b 

(Vjo  -  3)  ±  5 

"  1 

D  so  _  1)  ±  3 

B 

CAUCHY 

15 

(Vjo  -  1)  ±  5 

b 

(Vjo)  ±  1 

n 

D  so)  ±  3 

n 

(Vjo)  ±  5 

b 

A  SO  _  3(7)  ±  5(7 

b 

(Vjo)  ±  1(7 

B 

EXPONENTIAL 

15 

(Vj0)  ±  3a 

” 

(Vjo)  ±  5(7 

b 

(Vjo  +  3a)  ±  5(7 

b 

EMPIRICAL 

15 

(Vjo  -  3a)  ±  5<7 
(Vjo)  ±  1(7 
(Vjo)  ±  3(7 
(Vjo)  ±  5(7 
(Vjo  +  3(7)  ±  3a 

b 

b 

b 

b 

B 

ASYMMETRIC 
NOISE 
SV  t  GW 


NONE 


NONE 


NONE 


NONE 


TABLE  B2. 


DESIGN  (PRM)  AND  ESTIMATION  (NMLE) 


RESPONSE 

CURVE 

SAMPLE 

SIZE 

NORMAL 

6,12,15 

UNIFORM 

15 

cauoti' 

15 

EXPONENTIAL 

15 

EMPIRIC.AL 

15 

TABLE  B3. 
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TABLE  R4. 


DESIGN  (EQRC-DRM;  .AND  ESTIMATION  (NMLE) 


RESPONSE 

cunx 

SAMPLE 

SIZE 

NORMAL 

9,12,15 

UNIFORM 

15 

CAUCHY 

15 

EXPONENTIAL 

15 

EMPIRICAL 

15 

ZERO 
NOISE 
SV  &  GW 


NORMAL 
NOISE 
SV  &  GW 


.501  .411 

.495  .442 


ASYMMETRIC 
NOISE 
SV  &  GW 


.501  .452  .545  .522  .482  .547 

.422  .406  .507  .434  .377  .544 

.515  |  .458  .381  .541  .444  .363  .540 

.423  .404  .510  .484  .428  .490 

.456  .381  .541  .442  .371  .523 

.451  .389  .574  .454  .405  .563 


.498  .436  .391  .486 
.495  .445  .381  .526 


.544  .440  .401  .528  .455  .382  .525 


TABLE  B5. 


Dt-SIGN  VEQRC-ARM)  AND  ESTIMATION  (NMLE| 


RESPONSE 

SAMPLE 

ZERO 

NORMAL 

ASYMMETRIC 

CURVE 

SIZE 

NOISE 

NOISE 

NOISE 

SV  &  GW 

SV  &  GW 

SV  t  GW 

.503 

.459 

.559 

.516 

.475 

.557 

.411 

.453 

.410 

.530 

.439 

.373 

.533 

.416 

.391 

.510 

.447 

.383 

.547 

.452 

.382 

.544 

.454 

.367 

.476 

.420 

.409 

.490 

.479 

.429 

.469 

.432 

.385 

NORMAL 

9,12,15 

.525 

.470 

.375 

.537 

.449 

.381 

.514 

.431 

.597 

.455 

.394 

.573 

.464 

.411 

.587 

.485 

.412 

.463 

.396 

.355 

448 

.407 

.369 

.463 

.428 

.521 

.451 

.396 

.528 

.462 

.382 

.539 

.464 

.409 

.572 

.438 

.392 

.580 

.463 

.378 

.547 

.455 

.391 

.443 

.437 

.469 

.471 

.488 

.469 

UNIFORM 

15 

.445 

.452 

.468 

.479 

.482 

.462 

.308 

.329 

.339 

.307 

.391 

.401 

cauchy 

15 

.281 

.295 

.344 

.373 

.389 

.373 

.344 

.336 

.335 

.335 

EXPONENTIAL 

15 

.352 

.351 

.339 

.358 

.355 

.370 

17.240 

17.420 

17.470 

EMPIRICAL 

15 

18.590 

17.330 

Ml 

17.290 

17.370 
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TABLE  B6. 


DESIGN'  (LANGLIE)  AND  ESTIMATION  (NMLE) 


RESPONSE 

SAMPLE 

ZERO 

NORMAL 

ASYMMETRIC 

CUR\T 

SIZE 

NOISE 

NOISE 

NOISE 

SV  &  GW 

SV  &  GW 

SV  &  GW 

2.003 

2.000 

2.000 

2.001 

1.994 

BUSH 

1.951 

1.941 

.435 

.359 

.326 

.423 

.355 

.322 

.400 

.303 

.504 

.444 

.370 

.517 

.463 

.369 

.531 

.387 

.381 

.313 

.288 

.365 

.324 

.297 

.365 

.288 

NORMAL 

e.  12. 15 

.477 

.424 

.389 

.472 

.435 

.373 

.507 

.415 

.381 

.525 

.463 

.417 

.526 

.455 

.424 

.531 

.483 

.415 

.392 

.349 

.318 

.365 

.361 

.330 

.381 

.370 

.336 

.496 

.431 

.374 

.504 

.400 

.356 

.493 

.424 

.371 

.548 

.429 

.397 

.568 

.452 

.390 

.539 

.445 

.401 

.420 

.444 

.412 

.418 

.410 

.439 

UNIFORM 

15 

.402 

.382 

.416 

.406 

.402 

.414 

.313 

o-~ 

1 

.322 

.305 

.356 

.373 

CAUCHY 

15 

.232 

.245 

.346 

.327 

.409 

.391 

.354 

.346 

.286 

.279 

EXPONENTIAL 

15 

.349 

.358 

.410 

.418 

.387 

.378 

17.330 

17.030 

15.540 

EMPIRICAL 

15 

16.750 

15.720 

18.220 

17.810 

16.420 

17.020 
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TABLE  B7. 


DESIGN  (DRN1)  .AND  ESTIMATION  (AVR) 


RESPONSE 

SAMPLE 

ZERO 

NORMAL 

ASYMMETRIC 

CURVE 

SIZE 

NOISE 

NOISE 

NOISE 

S\' t  GW 

SV  k  GW 

SV  &  GW 

.5S7 

.486 

.440 

.526 

.476 

.530 

.480 

.435 

.512 

.412 

.378 

.513 

.432 

.379 

.537 

.432 

.394 

.502 

.465 

.390 

.533 

.453 

.398 

.565 

.457 

.387 

.486 

.430 

.431 

.497 

.487 

.455 

.471 

.482 

.446 

NORMAL 

r-  -  *>  %  r. 

5°° 

.459 

.399 

.525 

444 

.400 

.505 

.434 

.429 

.443 

.395 

.543 

.449 

' 

.556 

.475 

.395 

B  1 

.427 

.395 

.489 

.447 

K  Sfe 

.483 

.491 

.479 

Mill 

.439 

.375 

.495 

.434 

.520 

ESI 

.553 

.378 

.523 

.428 

.367 

.523 

m 

.369 

.431 

.463 

.460 

.455 

.455 

.451 

UNIFORM 

15 

.481 

.494 

.438 

.439 

.423 

.417 

.401 

.404 

422 

.401 

.483 

.577 

c.Auan' 

15 

.336 

.361 

.469 

.361 

.34  7 

.459 

.399 

.394 

.389 

.394 

EXPONENTIAL 

15 

.335 

.342 

.355 

.354 

.371 

.375 

EMPIRICAL 


15 


TABLE  B8. 


DESIGN  (.ARM)  AND  ESTIMATION  (AYR) 


RESPONSE 

CURVE 

SAMPLE 

SIZE 

NORMAL 

9,12.15 

UNIFORM 

15 

CAUCHY 

15 

EXPONENTIAL 

15 

EMPIRIC.AL 

15 

TABLE  B9. 


DESIGN  (EQRC'-DRM)  AND  ESTIMATION  (AYR) 


RESPONSE 

S.AMPLE 

ZERO 

NORMAL 

ASYMMETRIC 

cur\t: 

SIZE 

NOISE 

NOISE 

NOISE 

SV  £  GW 

SV  £  GW 

SV  £  GW 

.595 

.469 

.526 

.498 

.447 

.518 

.406 

.446 

.396 

.539 

.439 

.402 

.501 

.395 

.438 

.378 

.538 

.448 

.369 

.493 

.439 

.436 

.504 

.497 

.454 

.483 

.486 

.448 

NORMAL 

9,12.15 

.530 

.468 

.409 

mm 

.452 

.400 

.510 

.427 

.436 

.568 

.460 

.410 

EH 

.465 

422 

.550 

.486 

.418 

.499 

.440 

.421 

.493 

.460 

.439 

.487 

.491 

.481 

.502 

.163 

.431 

.498 

.460 

.405 

.532 

.462 

.552 

.440 

.406 

.459 

..395 

.528 

.445 

.389 

.423 

.452 

.473 

.474 

.489 

.480 

UNIFORM 

15 

.503 

.509 

.477 

.491 

.466 

.447 

.410 

.389 

.447 

.421 

.537 

.607 

CAUCHY 

15 

.349 

.358 

.494 

.397 

.440 

.508 

.381 

.382 

.389 

.394 

EXPONENTIAL 

15 

.387 

.385 

.373 

.385 

.366 

.381 

17.080 

17.140 

19.700 

12 

EMPIRICAL 

15 

18.940 

17.530 

15 

17.370 

17.330 
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TABLE  BIO. 


DESIGN  (EQRC-ARM)  AND  ESTIMATION  (AVR) 
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TABLE  Bll. 


DESIGN  (L.ANGLIE)  AND  ESTIMATION  (AYR) 


RESPONSE 

SAMPLE 

ZERO 

NORMAL 

ASYMMETRIC 

CURVE 

SIZE 

NOISE 

NOISE 

NOISE 

SV  £  GW 

SV  &  GW 

SV  £  GW 

2.003 

2.000 

2.000 

2.001 

2.000 

1.952 

1.945 

1.942 

.445 

.371 

.348 

.437 

.342 

.412 

.345 

.318 

.505 

.444 

.375 

.508 

.366 

.523 

.439 

.387 

.385 

.330 

.310 

.382 

.318 

.369 

.345 

.302 

NORMAL 

8.12,15 

.466 

.427 

.382 

.466 

.437 

.379 

.496 

.420 

.392 

.514 

.466 

.432 

.513 

.454 

.438 

.519 

.482 

.423 

.381 

.344 

.313 

.358 

.345 

.324 

.369 

.361 

.332 

.485 

.438 

.374 

.501 

.400 

.362 

.493 

.425 

.379 

.547 

.432 

.397 

.567 

.456 

.391 

.537 

.446 

.401 

.416 

.439 

.417 

.427 

.453 

.449 

UNIFORM 

15 

.386 

.365 

.420 

.405 

.402 

.417 

.283 

.263 

.321 

.295 

.371 

.392 

CAUCHY 

15 

.228 

.237 

.354 

.332 

.405 

.381 

.361 

.358 

.279 

.272 

EXPONENTIAL 

15 

.358 

.366 

.416 

.421 

.391 

.379 

17.190 

17.010 

15.090 

EMPIRICAL 

15 

hhb 

HHH 

16.250 

16.860 
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TABLE  B12. 


DESIGN  (DRM!  AND  ESTIMATION  (NEXT  STRESS) 
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TABLE  B13 


DESIGN  (ARM)  .AND  ESTIMATION  (NEXT  STRESS) 


RESPONSE 

CUR\"E 


SAMPLE 

SIZE 


ZERO 
NOISE 
SV&  GW 


NORMAL 
NOISE 
SV  &  GW 


56 


TABLE  B14. 


DESIGN  (EQRC-DRM)  .AND  ESTIMATION  (NEXT  STRESS) 


RESPONSE 

S.AMPLE 

— 

ZERO 

NORMAL 

ASYMMETRIC 

CURVE 

SIZE 

NOISE 

NOISE 

NOISE 

SY  £  GW 

SY  &  GW 

SV  &  GW 

.596 

.446 

.571 

.543 

.512 

.603 

.568 

.545 

.518 

.432 

.526 

.454 

.412 

.569 

.448 

.566 

.439 

.371 

.541 

.435 

.358 

.550 

.452 

.365 

.477 

.385 

.519 

.504 

.467 

.543 

.527 

NORMAL 

9.12.15 

.517 

.358 

.528 

.432 

.365 

.510 

.416 

.412 

.587 

422 

.616 

.493 

.499 

.621 

.506 

.454 

.483 

.401 

.362 

.517 

.471 

.442 

.529 

.555 

.569 

.537 

.474 

.427 

.537 

.487 

.421 

.584 

.499 

.420 

.626 

.521 

.486 

.637 

.550 

.485 

.617 

.553 

.460 

.430 

.437 

.442 

.474 

.512 

.486 

UNIFORM 

15 

.445 

.513 

.489 

.495 

.565 

.531 

.342 

.319 

.332 

.355 

.414 

.469 

CAUCHY 

15 

.300 

.362 

.411 

.363 

.401 

.431 

.348 

.336 

.339 

.410 

EXPONENTIAL 

15 

.374 

.387 

.440 

.465 

* 

.348 

.346 

16.740 

16.280 

17.510 

EMPIRICAL 

15 

19.070 

19.440 

20.440 

22.380 

17.380 

16.550 
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TABLE  B15. 


DESIGN  (EQRC-ARM)  AND  ESTIMATION  {NEXT  STRESS) 
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TABLE  B16. 


DESIGN  (LANG LIE)  AND  ESTIMATION  (NEXT  STRESS) 


RESPONSE 

SAMPLE 

ZERO 

NORMAL 

CLR\'E 

SIZE 

NOISE 

NOISE 

SV  &  GW 

SV  t  GW 

2.000 

2.000 

2.001 

1.995 

2.001 

.423 

.344 

.361 

.488 

.383 

.370 

.864 

1.255 

1.304 

.919 

1.332 

.381 

.311 

.308 

.378 

.351 

.356 

NORMAL 

9,12,15 

.960 

.694 

.775 

.922 

.760 

.801 

1.101 

1.097 

.941 

1.111 

1.126 

.982 

.451 

.442 

.405 

.445 

.462 

.432 

.759 

.947 

.731 

.788 

.938 

.699 

.966 

1.316 

.903 

1.024 

1.409 

.904 

1.297 

1.305 

.844 

.823 

1.000 

.972 

UNIFORM 

15 

.478 

.483 

.752 

.81) 

.914 

.885 

.497 

.476 

.536 

.521 

.764 

.712 

CAUCHY 

15 

.302 

.316 

.574 

.556 

.838 

.851 

.828 

.969 

.381 

.399 

EXPONENTIAL 

15 

.660 

.660 

.856 

.931 

1.300 

1.239 

48.890 

52.440 

EMPIRIC.NL 

15 

mi 

-  -  - 

46.810 

DISTRIBUTION  LIST 


No .  of 

No .  of 

Copies 

Organisation 

Copies 

Organisation 

12 

Administrator 

Defense  Technical  Info.  Center 

ATTN:  DTIC-DDA 

Cameron  Station 

Alexandria,  VA  22504-6145 

1 

Commander 

US  Army  Armament,  Munitions 
and  Chemical  Command 

ATTN:  SMCAR-ESP-L 

Rock  Island,  IL  61299 

1 

HQDA  (SARD-TR) 

Washington,  DC  20510 

1 

Commander 

US  Army  Aviation  Systems 
Command 

1 

Commander 

US  Army  Materiel  Command 

ATTN:  'aMCDRA-ST 

5001  Eisenhower  Avenue 

ATTN:  AMSAV-DACL 

4500  Goodfellow  Blvd. 

St.  Louis,  MO  65120 

Alexandria,  VA  22555-0001 

1 

Director 

US  Army  Aviation  Research 

1 

Commander 

US  Army  Laboratory  Command 

ATTN:  AMSLC-TD 

Aaelphi,  MD  20785-114=; 

and  Technology  Activity 
Ames  Research  Center 

Moffett  Field,  CA  94055-1099 

1  Commander 

Armament  R&D  Center 
US  Army  AMCCOM 
ATTN:  SMC AR- MS  I 

Picatinny  Arsenal,  NJ  07806-5000 


Commander 

Armament  RSD  Center 

US  Army  AMCCOM 

ATTN:  SMCAR-TDC 

Picatinny  Arsenal,  NJ  07806-5000 

1 

Commander 

US  Army  Missile  Command 

Director 

ATTN:  AMSMI-AS 

Benet  Weapons  Laboratory 

Armament  RSD  Center 

Redstone  Arsenal,  AL  55898-5010 

US  Army  AMCCOM 

1 

Commander 

ATTN:  SMCAR-LCB-TL 

US  Army  Tank  Automotive 

Watervliet,  NY  12189 

Command 

ATTN:  AMSTA-DI 

Warren,  MI  48090 

61 


DISTRIBUTION  LIST 


No.  of 

Copies  Organi cation 

1  Director 

US  Army  TRADOC  Analysis 
Command 

ATTN:  ATAA-SL 

White  Sands  Missile  Range,  NM 
88002 

1  Commandant 

US  Army  Infantry  School 
ATTN:  ATSH-CD-CSO-OR 
Fort  Benning,  GA  31905 

1  AFWL/SUL 

Kirtland  AFB,  NM  87117 

1  Air  force  Armament  Laboratory 

ATTN:  AFATL/DLODL 
Eglin  AFB,  FL  32542-5000 

1  US  Army  Concepts  Analysis  Agency 

Models  Application  Division 
Chief,  Math  Stat  Team 
ATTN:  CSCA-RSA,  Carl  B.  Bates 
8120  Woodmont  Avenue 
Bethesda ,  MD  20814-2797 

1  TRAC  Director  of  Research 

ATTN:  ATRC-RD,  D.  H.  McCoy 
White  Sands  Missile  Range,  NM 
88002-5502 

1  Director 

USATEC  (P) 

ATTN:  Dr.  Marion  R.  Bryson 
Fort  Ord,  CA  95941-7000 


No.  of 
Copies 


1 


2 


1 


Organication 

University  of  Delaware 
Department  of  Mathematical 
Statistics 

ATTN:  Prof.  Henry  B.  Tingey 

Newark,  DE  19716 

Dir,  USAMSAA 
ATTN:  AMXSY-D 

AMXSY-MP ,  H.  Cohen 

Cdr ,  USATECOM 
ATTN:  AMSTE-TO-F 

Cdr,  CRDEC ,  AMCC0M 
ATTN:  SMCCR-RSP-A 
SMCCR-MU 
SMCCR-SPS-IL 


62 


USER  EVALUATION  SHEET/CHANGE  OF  ADDRESS 


Thi?  ]^brratory  undertakes  a  continuing  effort  to  improve  the  quality  of  the  reports  it  publishes.  Your 
comments /answers  below  will  aid  us  in  our  efforts. 

1.  Does  this  report  satisfy  a  need?  (Comment  or.  purpose,  related  project,  or  other  area  cf  interest  for  which 
the  report  will  be  used.)  _ ____ _ _ _ 


2.  How,  specifically,  is  the  report  being  used?  (Information  source,  design  data,  procedure,  source  cf  ideas, 
etc.)  _ _ 


3.  Ha*?  the  information  in  this  report  led  to  any  quantitative  savings  as  far  as  man-hours  or  dollars  savec, 
operating  costs  avoided,  or  efficiencies  achieved,  etc?  If  so,  please  elaborate.  _ 


4.  General  Comments.  What  do  you  think  should  be  changed  to  improve  future  reports?  (Indicate  changes  tc 
organization,  technical  content,  format,  etc.)  _  _  _ 


BRL  Report 
Check  here 
Check  here 
Current 


Number  _  Division  Symbol  _ 

if  desire  to  be  removed  from  distribution  list, 
for  address  change.  _ 

address:  Organization  _ 

Address 


-FOLD  AND  TAPE  CLOSED 


Director 

U.S.  Army  Ballistic  Resea 
ATTN:  S LCBR-DD-T (NEI ) 
Aberdeen  Proving  Ground, 

OEElClAl  BUSINESS 

PENAUT  EO»  private  USE  1300 


U.S.  Army  Ballistic  Research  Laboratory 
ATTN:  S LCBR-DD-T (NEI) 

Aberdeen  1  roving  Ground,  MD  21005-9989 


rch  Laboratory 
MD  21005-5066 


BUSINESS  REPLY  LABEL 

MRS!  Cl  ASS  PERMIT  NO  \  2062  WASHINGTON  D  C 


POSTAGE  Wltl  BE  PAID  BY  DEPARTMENT  Of  THE  ARMY 


Di rector 


NO  POSTAGE 
NECESSARY 
If  MAILED 
IN  THE 

UNITED  STATES 


